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Abstract 

The synapses of real neural systems seem to have delays. Therefore, it is worthwhile 
to analyze associative memory models with delayed synapses. Thus, a sequential asso- 
ciative memory model with delayed synapses is discussed, where a discrete synchronous 
updating rule and a correlation learning rule are employed. Its dynamic properties 
are analyzed by the statistical neurodynamics. In this paper, we first re-derive the 
Yanai-Kim theory, which involves macrodynamical equations for the dynamics of the 
network with serial delay elements. Since their theory needs a computational complex- 
ity of 0{LH) to obtain the macroscopic state at time step t where L is the length of 
delay, it is intractable to discuss the macroscopic properties for a large L limit. Thus, 
we derive steady state equations using the discrete Fourier transformation, where the 
computational complexity does not formally depend on L. We show that the storage 
capacity ac is in proportion to the delay length L with a large L limit, and the pro- 
portion constant is 0.195, i.e., ac = 0.195L. These results are supported by computer 
simulations. 



*This research was partially supported by the Ministry of Education, Science, Sports and Culture, 
Japan, Grant-in- Aid for Scientific Research, 13780303. 

^Corresponding author. E-mail:miyoshi@kobe-kosen. ac.jp 



1 



S.Miyoshi, H.-F.Yanai, and M.Okada 2 

Key words: sequential associative memory, neural network, delay, statistical neurody- 
namics 



1 Introduction 

Associative memory models of neural network can be mainly classified into two types 
(Okada, 1996; Fukushima, 1973; Miyoslii & Okada, 2000). The first is the auto asso- 
ciative memory model where memory patterns are stored as equilibrium states of the 
network. The second type is the sequence processing model, which stores the sequence 
of memory patterns. 

As a learning algorithm for storing memory patterns, the correlation learning algo- 
rithm based on Hebb's rule is well known. The storage capacity, that is, how many 
memory patterns can be stably stored with respect to the number of neurons, is the 
one of the most important properties of associative memory models. Hopfield showed 
that the storage capacity of the auto associative memory model using correlation learn- 
ing is about 0.15 by computer simulation (Hopfield, 1982). On the other hand, many 
theoretical analyses have been done on the correlation learning type associative mem- 
ory model (Okada, 1996). As typical analytical methods, there are the rephca method 
(Sherrington & Kirkpatrick, 1975; Amit et al., 1985a; Amit et al., 1985b) and the SC- 
SNA (Shiino & Fukai, 1992) for the equilibrium state of the auto associative memory 
models. There is also the statistical neurodynamics (Amari & Maginu, 1988) for han- 
dling retrieval dynamics. By these theories, it became clear that the storage capacity of 
the auto-associative memory model is 0.138 and that of the sequence processing model 
is 0.269 (Amari, 1988; Diiring et al., 1998; Kawamura & Okada, 2002). Furthermore, it 
has become well-known that the analysis of the dynamics for the auto-associative model 
is more difficult than that for the sequence processing model. However, Okada (Okada, 
1995) succeeded in explaining the dynamics of the retrieval process quantitatively by 
extending the statistical neurodynamics (Amari & Maginu, 1988). 

On the other hand, the synapses of real neural systems seem to have delays. There- 
fore, it is very important to analyze the associative memory model with delayed synapses. 
Computer simulation is a powerful method for investigating the properties of the neural 
network. However, there is a limit on the number of neurons. In particular, computer 
simulation for a network that has large delay steps is realistically impossible consid- 
ering the required amount of calculation and memory. Therefore, the theoretical and 
analytical approach is indispensable to research on delayed networks. 

A neural network in which each neuron has delay elements (Fukushima, 1973; Yanai 
& Kim, 1993; Miyoshi & Nakayama, 1995) was analyzed by Yanai and Kim (Yanai & 
Kim, 1993). They analyzed the delayed network using the statistical neurodynamics 
(Amari & Maginu, 1988; Amari, 1988) and derived macrodynamical equations for the 
dynamics. Their theory closely agrees with the results of computer simulation. 

In this paper, after defining the model, the Yanai-Kim theory is re-derived by using 
the statistical neurodynamics (Miyoshi & Okada, 2002). We show that their macrody- 
namical equations make clear that the dynamics of the network and the phase transition 
points change with the initial conditions. The Yanai-Kim theory needs a computational 
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complexity of 0{LH) to obtain the macrodynamics, where L and t are the length of 
delay and the time step, respectively. This means that it is indispensable to discuss 
the macroscopic properties for a large L limit. Thus, we derive the macroscopic steady 
state equations by using the discrete Fourier transformation. Using the derived steady 
state equations, the storage capacity can be quantitatively discussed even for a large 
L limit. Then, it becomes clear that the phase transition point calculated from the 
macroscopic steady state equations agrees with the phase transition point obtained by 
time-dependent calculation with a sufficient number of time steps from the optimum 
initial condition. Furthermore, it becomes clear that in the case of large delay length 
L, the storage capacity is in proportion to the length and the proportion constant is 
0.195. These results are supported by computer simulation. 



2 Model of delayed network 

The structure of the delayed network discussed in this paper is shown in Figure |l[ 
The network has neurons, and L — 1 serial delay elements are connected to each 
neuron. All neurons as well as all delay elements have synaptic connections with all 
neurons. In this neural network, all neurons and all delay elements change their states 
simultaneously. That is, this network employs a discrete synchronous updating rule. 
The output of each neuron is determined as 

= F{u^, (1) 

L-l N 

< = EE -44-'. P) 

where x\ denotes the output of the zth neuron at time t, and Jl^ denotes the connection 
weight from the /th delay elements of the jth neuron to the ith neuron. F {■) is the 
sign function defined as 

Fi,n)^.Mu) = {^_\lll (3) 

Now, let's consider how to store the sequence of aN memory patterns, — > — >■ 
•■■ ^'^ ^ Here, a and aN are the loading rate and the length of the 

sequence, respectively. Each component is assumed to be an independent random 
variable that takes a value of either +1 or —1 according to the following probabilities, 

prob[er = ±i] = ^. (4) 

We adopt the following learning method using the correlation learning, 

4 = ;^ Em;. (5) 

where q is the strength of the /th delay step. 
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Figure 1: Structure of delayed network. 



Correlation learning is an algorithm based on Hebb's rule. It is inferior to the error 
correcting learning in terms of storage capacity. However, as seen from eqn (|^), when 
adding new patterns, it is not necessary to again learn all patterns that were stored in 
the past. Furthermore, correlation learning has been analyzed by many researchers due 
to its simplicity. 

3 Dynamical behaviors of macroscopic order pa- 
rameters by statistical neurodynamics and dis- 
cussion 

As mentioned above, a neural network in which each neuron has delay elements (Fukushima, 
1973; Yanai & Kim, 1993; Miyoshi & Nakayama, 1995) was analyzed by Yanai and Kim 
(Yanai & Kim, 1993). In this section, the Yanai-Kim theory is re-derived by using the 
statistical neurodynamics (Miyoshi & Okada, 2002). Using the re-derived theory, we 
discuss the dynamical behaviors of a delayed network. Furthermore, the relationship 
between the phase transition point and the initial conditions of the delayed network is 
discussed. 

In the case of a small loading rate a, if a state close to one or some of the patterns 
stored as a sequence are given to the network, the stored sequence of memory patterns 
is retrieved. However, when loading rate a increases, the memory is broken at a certain 
a. That is, even if a state close to one or some of the patterns stored as a sequence 
is given to the network, the state of the network tends to leave the stored sequence of 
memory patterns. Moreover, even if one or some of the patterns themselves are given 
to the network, the state of the network tends to leave the stored sequence of memory 
patterns. This phenomenon, that is, the memory suddenly becoming unstable at a 
critical loading rate, can be considered a kind of phase transition. 

We define the overlaps or direction cosine between a state = (x*) appearing in a 
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recall process at time t and an embedded pattern = (^f ) as 



1 ^ 



i=l 

By using this definition, when the state of the network at time t and the ^th pattern 
agree perfectly, the overlap rrif^ is equal to unity. When they have no correlation, the 
overlap is equal to zero. Therefore, the overlap provides a way to measure recall 
quality. 

Amari and Maginu (Amari & Maginu, 1988) proposed the statistical neurodynamics. 
This analytical method handles the dynamical behavior of the recurrent neural network 
macroscopically, where cross-talk noise is regarded as a Gaussian random variable with 
a mean of zero and a time-dependent variance of a^. They then derived recursive 
relations for the variance and the overlap. 

Using eqns @, and (H), the total input of the zth neuron at time t is given as 

L-1 N 

< = EE 44" (7) 

= s^e' + 4, (8) 
= Y.^i<-\. (9) 

1=0 

< = E^'E^r'<-V (10) 

Z=0 uj^t 

The first term in eqn (P) is the signal that is useful for recall, while the second term 
is cross-talk noise which prevents from being recalled. This procedure is called a 
signal-to-noise analysis. 

The overlap m* can be expressed as 



ml 



1 ^ 

i^E^r^ (11) 

■;=1 

L-1 

+ [/,^c,m;rj;r\, (12) 

l'=0 

1 ^ f^~^ ^ c 

TV E ^« ^ ( E E ]v 

i=l \l=0 j=l 

X E r^^e^.-'-^), (13) 

1 ^ N ^ 

N E^' EE 

i=l \l=o j=l 

X E r^+'g^-'-'V (14) 
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Taking into account the correlation in the cross-talk noise zj, we have derived the 
following macrodynamical equations using eqns (p])-(p!4D (see Appendix). 



L-l L-l 

^^CiCi:Vt-l,t-l', (15) 
1=0 l'=0 

Vt-i,t-i' = aSi^i' 

L-l L-l 

X CkCk'Vt-l-k-l,t-l'-k'-l 
k=0 k'=0 

+ a {ci-v-iUt-v + cv-i-iUt-i) , (16) 
2 1 / fs*-i^'^ 



L-l 



^ciuit^i, (18) 



(=0 



where denotes m^. If t < 0, = and Ut = 0. If A; < 0, Ck = 0. If either k < 
or k' < 0, Vk^k' = 0. The expression erf (x) = ^ exp (— m^) (iu denotes the error 
function. 

Various cases can be considered as the initial condition of the network. For example, 
in the case of the one step set initial condition, only the states of neurons are set 
explicitly and those of delay elements are all zero. Then mo = mi^it is the only initial 
condition that is set explicitly and fo,o = On the other hand, as one more extreme 
case, the all steps set initial condition can also be considered, where the states of all 
neurons and all delay elements are set to be close to the stored pattern sequences. 
In this case, mi = minit, / = 0, ■ ■ ■ , L — 1 and vi^i = a, I = 0, ■ ■ ■ , L — 1. In the 
case where all neurons and all delay elements are set to minit = 1, that is, m; = 
1, / = 0, ■ ■ ■ , L — 1, the maximum information for recalling the stored sequence of 
memory patterns is given. Therefore, this condition can be called the optimum initial 
condition. Furthermore, the storage capacity ac is defined as the critical loading rate 
where recalling becomes unstable under the optimum initial condition. We note that 
the derived macrodynamical equations and the Yanai-Kim theory (Yanai & Kim, 1993) 
coincide with each other. 

Some examples of the dynamical behaviors of recall processes by the above theory 
and computer simulations are shown in Figures In these figures, the abscissa is 
time t, and the ordinate is the overlap mt- These are the results when the all steps set 
initial conditions are given. Figures and ^ are the results of 30-step time-dependent 
theoretical calculation by eqns ([T5| ) - ([T9|) using various initial overlaps. The details of 
the computer simulations shown in Figures ^ and ^ are as follows. First, a sequence 
of random patterns are generated. The length of the sequence is aN = 1000. Each 
pattern is a vector in which the dimension is = 2000. Therefore, the loading rate 
is ct = 0.5. Each element of the pattern vectors takes a value of either -|-1 or —1 with 
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Figure 2: Dynamical behaviors of recall process {L — 2,a — 0.5: theory). 
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Figure 3: Dynamical behaviors of recall process {L — 2,a — 0.5: computer simulation 
with TV = 2000). 
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probability ^. Next, the sequence is stored by correlation learning in two networks of 
which the number of neurons is = 2000 and the lengths of delay L are two and 
three. Then, various pattern sequences are generated where the initial overlaps with 
the Ist-Lth patterns stored are 0.0 — 1.0. These are given as the initial state of the 
network. After that, 30-step calculation is carried out by using the updating rule of the 
network, that is, eqns 

According to these figures, the dynamical behaviors of the overlaps obtained by the 
re-derived theory are in good agreement with those obtained by computer simulation. 
Figure |^ shows the dynamical behaviors of the overlaps in the case of L = 2. When 
the initial overlaps are 0.1 — 0.7, the overlaps somewhat increase at the first time step. 
However, the recall processes eventually fail regardless of the initial overlaps. This fact 
indicates that the storage capacity ac in the case of L = 2 is smaller than 0.5. On 
the other hand. Figure ^ shows the dynamical behaviors of the overlaps in the case of 
L = 3. When the initial overlap is 1.0, at the first time step the overlap somewhat 
decreases. However, then the overlap tends to a value near 1.0. This fact indicates that 
the storage capacity ac in the case of L = 3 is larger than 0.5. 



1 
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^ 0.6 
o 0.4 
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Figure 4: Dynamical behaviors of recall process (L = 3, a = 0.5: theory). 

Figure ^ shows the relationship between the loading rates a and the steady state 
overlaps moo obtained by time-dependent theoretical calculations with a sufficient num- 
ber of steps. Figure |^ shows the results of computer simulations under the same condi- 
tions as the theoretical calculations. In each figure, "(1)" and "(A)" denote the one step 
set initial condition and the all steps initial condition, respectively. In the computer 
simulations, the number of neurons is A^ = 500. The computer simulations have been 
carried out under five conditions : L = 1, L = 3 from the one step set initial condition, 
L = 3 from the all steps initial condition, L = 10 from the one step set initial condition, 
and L = 10 from the all steps initial condition. Eleven simulations have been carried 
out at various loading rates a under each condition. Here, the initial overlap mjnit = 1 
in all casesDTherefore, the all steps set initial condition is equivalent to the optimum 
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Figure 5: Dynamical behaviors of recall process {L = 3,a = 0.5: computer simulation 
with = 2000). 

initial condition as described above. In Figure 0, data points • , o , ■ , □ , * indicate 
the medians of the 6th largest values in the eleven trials. Error bars indicate the third 
and the ninth largest values in the eleven trials. 

These figures show that the steady states obtained by the re-derived theory are in 
good agreement with those obtained by computer simulation. In the case of L = 3, the 
difference between the phase transition point under the one step set initial condition 
and that under the optimum initial condition is small. However, in the case of L = 10, 
the difference is large. That is, the influence of the initial condition of the network 
increases as the length of delay L increases. We note that in the range between the phase 
transition point under the one step set initial condition and that under the optimum 
initial condition, there is the phenomenon that the attractor cannot be recalled from the 
former initial condition, although the sequence of memory patterns is certainly stored. 

Figure ^ shows the relationship between the length of delay L and the critical load- 
ing rate where the phase transition occurs. This theory (Yanai-Kim theory) needs a 
computational complexity of 0{LH) to obtain the macrodynamics, where L and t are 
the length of delay and the time step, respectively. Therefore, in this method, it is in- 
tractable to investigate the critical loading rate in the case of such a large delay L. Here, 
the results in the cases of L = 1, 2, ■ ■ ■ , 10 are shown. This figure shows the following. 
When the initial conditions are optimum, the relationship between the length of delay 
L and the critical loading rate, that is, the storage capacity, seems to be almost linear. 
It is assumed that when the length of delay L further increases, this tendency would 
continue. These characteristics are analyzed in the next section. On the other hand, 
in the case of the one step set initial condition, the critical loading rate is not linear 
with the length of delay L but saturated. However, the reason for this phenomenon is 
the lack of initial information. We note that this saturation doesn't imply the essential 
saturation of the storage capacity of the delayed networks. 
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Figure 6: Relationship between loading rate a and overlap m. "(1)" and "(A)" indicate 
one step set initial condition and all steps set initial condition, respectively (theory). 
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Figure 7: Relationship between loading rate a and overlap m. "(1)" and "(A)" indicate 
one step set initial condition and all steps set initial condition, respectively (computer 
simulation with = 500). 
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Figure 8: Length of delay L and loading rate of phase transitions. Solid line and 
dashed hne show results for one step set initial condition and those for all steps set 
initial condition, respectively. 



4 Macroscopic steady state analysis by discrete Fourier 
transformation and discussion 

The Yanai-Kim theory re-derived in the previous section, that is, the macrodynamical 
equations obtained by the statistical neurodynamics, needs a computational complexity 
of 0{LH) to obtain the macrodynamics shown in eqns (15) and (0), where L and t 



are the length of delay and the time step, respectively. Therefore, in this method, 
it is intractable to investigate the critical loading rate for a large L limit, that is, 
the asymptotic behavior of the storage capacity in the large L limit. Thus, in this 
section, the Yanai-Kim theory in a steady state is considered. After that, we derive the 
macroscopic steady state equations of the delayed network by using the discrete Fourier 
transformation, where the computational complexity does not formally depend on L. 
Furthermore, the storage capacity is analytically discussed for a large L by solving the 
derived equations numerically. 

For simplicity, let us assume Q = 1, / = 0, ■ ■ ■ , L — 1. In a steady state, at, Ut and 
rut in eqns (|l^)-(^) can be expressed as a, U and m, respectively. In addition, vt-i^t-v 
can be expressed as vi^ii because of the parallel symmetry in terms of v. Therefore, 
modifying eqns (|T^) and (|TB|), we obtain 

L-l 

= Yl iL-\n\)v{n), (20) 

n=l-L 

V (n) = a6n,o 

L-l 

+ Yl iL-\i\)v{n-i) + aUd{n), (21) 

i=l-L 
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d(n) = I ^' I'^l - ^'^'---'-^ C22) 
[0, otherwise ' 

where n = I — I' ,i = k — k' , v{n) denotes Vn and 6 is Kronecker's delta. 

Using the discrete Fourier transformation, we derive the general term of v{n), which 
is expressed by the recurrence formula in eqn (21). Applying the discrete Fourier 
transformation to eqns (^) and (p2D, we obtain 

L-l 



V{r) = a + ^y^ ^ (L- |2|)\/(r)e-^'2"^ +at/D(r), (23) 

i=l-L 

T 

D{r) = J2 d{n)e~^^''^ 

n=-T 
L 

I g -'^'^2T+1 _|_ gJ^'^2T+l 



n=l 
L 



n=l 



where V(r) and D(r) are the discrete Fourier transformation of v{n) and d{n)^ respec- 
tively. 

Solving eqns (E3) and (E4|) in terms of V{r\ we obtain 



Since the inverse discrete Fourier transformation of this equation equals f (r?,), we 
obtain 



T 

i 



L_ 5^ y(r)e^-^-^. (26) 



2T+ _ 

r=-T 



Substituting eqn (^) into eqn (pOD, we obtain 

^ T L-l 



(27) 



= > V [r] y (L- |n|)e^^"2T+i 

' ~ T=-T n=l-L 



Substituting eqn ( |25D into eqn ( p7[ ) and calculating the equation in the large T limit, 
we can express cr^ as the form using a simple integral like eqn (28). 

As a result, we can obtain the steady state equations in terms of the macroscopic 
variables of the network as eqns (^)- (|3T|) . 

^2 _ /"^ « [(1 - U) sin(7ra;) + [/sin {(2L + 1) nx}] [1 - cos{2Lttx)] 

^ ~ sin(^x) [2sin2(7ra;) - {1 ~ cos(2L^a:)}] ^ ^ ^ 
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s = mL (30) 
m = erf ( J (31) 



Though the derived macroscopic steady state equations include a simple integral, 
their computational complexity does not formally depend on L. Therefore, we can 
easily perform numerical calculations for a large L. 

Figure ^ shows the relationship between the loading rate a and the overlap m, which 
is obtained by solving these equations numerically. 

Comparing Figure ^ and ^ especially considering the case of L = 1,3, 10, we can 
see that the overlaps obtained by the macroscopic steady state equations in this section 
agree with those obtained by the dynamical calculations having sufficient time steps 
from the optimum initial conditions in the previous section. In other words, the phase 
transition points obtained by the macroscopic steady state equations derived in this 
section agree with the storage capacity, that is, the phase transition points under the 
optimum initial condition. This fact shows that the solution to the dynamical equa- 
tions from the optimum initial condition and the solution to the steady state equations 
support each other. 



Figure |T0| shows the relationship between the length of delay L and the storage ca- 
pacity ac- From this figure, we can see that the storage capacity increases in proportion 
to the length of delay L with a large L limit and the proportion constant is 0.195. That 
is, the storage capacity of the delayed network ac equals 0.195L when the length of 
delay L is large. Though the result that the storage capacity of the delayed network is 
in proportion to the length of delay L is not nontrivial, the fact that this result has been 
proven analytically is significant. The proportion constant 0.195 is the mathematically 
significant number as the limit of the delayed network's storage capacity. 



5 Conclusions 

We analyzed sequential associative memory models with delayed synapses. First, we 
re-derived the Yanai-Kim theory, which involves the macrodynamical equations for net- 
works with serial delay elements. Since their theory needs a computational complexity 
of 0{LH) to obtain the macroscopic state at time step t where L is the length of de- 
lay, it is intractable to discuss the macroscopic properties for a large L limit. Thus, 
we derived steady state equations using the discrete Fourier transformation, where the 
complexity does not formally depend on L. We showed that the storage capacity ac 
is in proportion to delay length L with a large L limit, and the proportion constant is 
0.195, i.e., ac = 0.195L. These results were supported by computer simulation. 
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Figure 9: Relationship between loading rate a and overlap m. These lines are obtained 
by solving steady state equations numerically. 
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Figure 10: Relationship between length of delay L and storage capacity ac- This line is 
obtained by solving the steady state equations numerically. Storage capacity is 0.195L 
with large L limit. 
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Appendix A. Derivations of the macro dynamical equa- 
tions of delayed network 



Derivations of the macrodynamical equations of delayed network eqns (p!5|)-([T9|) dis- 
cussed in Section ^ are given here. Using eqns ([T0|) and (|12D, we obtain 

zl = za + zb, (32) 

= E^^E^^^^Er^"^'^"^ (33) 

= (34) 

/=0 /i^t l'=0 

where x*"'^^"''' is the variable obtained by removing the influence of from x^f^- 
Using eqns (P^)-(|5^), we obtain 

E[z^ = 0, (35) 

^(4)'] (36) 
= E[z\ + zI + 2zaZb]. (37) 
Transforming z\, z]^, zaZb with consideration given to their correlation, we obtain 

E[zl] = aJ2cl (38) 
1=0 

L-l L-1 L-1 L-1 
^^t 1=0 l'=0 k=0 k'=0 

X t/*-.t/.-.m;;Jr-t\<irir\ (39) 

L-1 L-1 

E [2zaZb] = a XI X] 

1=0 l'=0 



al = E 



X {ci-v-iUt-u + cv-i-iUt-i) , (40) 



where 



at = a 



vt-i,t-i' = J2 minimis, . (41) 
Using eqns (|37D-(^0D, we obtain 

L-1 
1=0 

L-1 L-1 L-1 L-1 

+ X] X] X] X] QCi'CkCk' 
1=0 l'=0 k=0 k'=0 

X Ut-iUt-i'Vt-i-k-i,t-i'-k'-i 

L-1 L-1 

+ " X] 5Z i<^i-i'-iUt-i' + cv_i_iUt-i) . (42) 

1=0 l'=0 
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Using eqns (p^Of ) and (0), we obtain 

L-l L-l 



= XI 5Z cici>vt-i,t-i' ■ (43) 



,2 _ 

1=0 l'=0 

Comparing eqns (^) and ( ^31) as identical equations regarding qq/, we obtain 

L-l L-l 
k=0 k'=0 

+ a {ci-v-iUt-v + cv-i-iUt^i) , (44) 
where 5 is Kronecker's delta. Using eqn ([1^), we obtain 

TV /L-l Af 



Cl 



i=l \l=0 j=l 

E [F' (m*^'^))] (46) 

E [F' [u')] (47) 

e-^ « F' (ti*) » (48) 

< F (?/*) > (49) 

where is the variable obtained by removing the influence of from u^. -C ■ 3> 
stands for the average over pattern ^. 

As a result, we can obtain the macrodynamical equations for overlap m, that is, 
eqns (|1)-(I1). 



2n 

1 r dz .2 



J V2n 



e 2 z 
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